REINFORCEMENT LEARNING

Neural Math Word Problem Solver with Reinforcement Learning (C18-1)

Challenge:	Current approaches to visual dialog learning involve an end-to-end framework that maps the multi-modal context to a deep vector and in order to decode a natural dialog response.
Approach:	They propose a framework that trains a RL policy for image guessing and a seq2seq model to improve dialog quality.
Outcome:	The proposed framework achieves state-of-the-art performance on a guessWhich task . it can be applied to a wide range of tasks including assisting blind people .

Copied to clipboard

Danqing Huang, Jing Liu, Chin-Yew Lin, Jian Yin

Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning (2022.coling-1)

Challenge:	Existing models for solving math word problems rely on predefined rules or feature engineering.
Approach:	They propose to incorporate copy and alignment mechanism into the sequence-to-sequence model to address two shortcomings . they use model output as a feature and incorporate it into the feature-based model to explore the effectiveness .
Outcome:	The proposed model outperforms the state-of-the-art models on the problem solving task.

Copied to clipboard

Atsumoto Ohashi, Ryuichiro Higashinaka

Few-Shot Structured Policy Learning for Multi-Domain and Multi-Task Dialogues (2023.findings-eacl)

Challenge:	In task-oriented dialogue systems, the role of the natural language generation component is to convert a system's intentions, called dialogue acts (DAs), into natural language utterances and to convey DAs accurately to users.
Approach:	They propose a method for Adaptive Natural language generation for Task-Oriented dialogue via Reinforcement learning that incorporates a natural language understanding module into the objective function of RL.
Outcome:	The proposed method generates adaptive utterances against speech recognition errors and the different vocabulary levels of users in a multi-world task-oriented dialogue system.

Copied to clipboard

Thibault Cordier, Tanguy Urvoy, Fabrice Lefèvre, Lina M. Rojas Barahona

Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings (2026.tacl-1)

Challenge:	Reinforcement learning is widely adopted to model dialogue managers in task-oriented dialogues, but the user simulator provided by state-of-the-art dialogue frameworks are only rough approximations of human behaviour.
Approach:	They propose to use structured policies to improve sample efficiency when learning on multi-domain and multi-task environments.
Outcome:	The proposed policies improve sample efficiency and performance on multi-domain and multi-task environments.

Copied to clipboard

Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick Fernandes, André F. T. Martins

Machine Translation

Challenge:	Reinforcement learning (RL) is an effective and robust method for training neural machine translation systems.
Approach:	They propose a method that leverages fine-grained, token-level quality assessments . they use a state-of-the-art quality estimation system as their token- level reward model .
Outcome:	The proposed approach leverages fine-grained, token-level quality assessments along with error severity levels to improve translation quality.

LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization (2020.coling-main)

Copied to clipboard

Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-chin Lin, Marco Moresi, Carel van Niekerk, Milica Gasic

Fine-Tuning Language Models with Reward Learning on Policy (2024.naacl-long)

Challenge:	Reinforcement learning (RL) can be used to steer a conversation towards successful task completion.
Approach:	They propose to use latent latent variables to shape latent variable distributions . they use response auto-encoding as auxiliary task to capture generative factors .
Outcome:	The proposed approach yields a more action-characterized latent representations . the proposed approach achieves state-of-the-art success rates .

Copied to clipboard

Hao Lang, Fei Huang, Yongbin Li

RL with KL penalties is better viewed as Bayesian inference (2022.findings-emnlp)

Challenge:	Reinforcement learning from human feedback (RLHF) is an effective approach to align large language models (LLMs) to human preferences.
Approach:	They propose a framework that refines a reward model using policy samples to keep it on-distribution.
Outcome:	The proposed framework outperforms the state-of-the-art on three benchmark datasets showing that it can learn robust representations of policy samples.

Copied to clipboard

Tomasz Korbak, Ethan Perez, Christopher Buckley

IterAlign: Iterative Constitutional Alignment of Large Language Models (2024.naacl-long)

Challenge:	Reinforcement learning (RL) is used in fine-tuning large language models to penalize them for undesirable features of generated sequences.
Approach:	They analyze challenges associated with treating a language model as an RL policy . they find that RL is equivalent to variational inference: approximating a Bayesian posterior .
Outcome:	The proposed approach is flawed because it turns the LM into a degenerate distribution, the authors show . they show that the proposed approach avoids the distribution collapse problem and offers a first-principles derivation for its objective.

Copied to clipboard

Xiusi Chen, Hongzhi Wen, Sreyashi Nag, Chen Luo, Qingyu Yin, Ruirui Li, Zheng Li, Wei Wang

VEG: Verbal 𝜖-greedy for Semantic Exploration in Multi-Turn RL Agents (2026.acl-industry)

Challenge:	Empirical results show that iterAlign improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to 13.5% in harmlessness.
Approach:	They propose a data-driven constitution discovery and self-alignment framework called IterAlign to overcome these drawbacks by leveraging red teaming to uncover weaknesses of an LLM.
Outcome:	Empirical results show that iterAlign improves truthfulness, helpfulness, harmlessness and honesty by up to 13.5%.

Copied to clipboard

Yongchang Hao, Jie Hao, Yongsheng Mei, Ze Ye, Junyi Chai, Bin Guo, Benjamin Z. Yao, Chenlei Guo, Lili Mou

Challenge:	Standard RL approaches suffer from reward sparsity and mode-seeking behavior . lack of diversity hinders exploration necessary for optimal learning .
Approach:	They propose a framework that leverages external feedback as a dynamic control variable to explicitly balance exploration and exploitation within the semantic space.
Outcome:	Experiments on Tau Bench and SearchQA show that the proposed framework outperforms standard RL baselines.

Language Model Adaption for Reinforcement Learning with Natural Language Action Space (2024.acl-long)

Copied to clipboard

Jiangxing Wang, Jiachen Li, Xiao Han, Deheng Ye, Zongqing Lu

From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization (2026.acl-long)

Challenge:	Previous research has focused on reducing the size of the natural language action space due to the combinatorial nature of the language.
Approach:	They propose mutual-information regularized policy optimization to reduce the action space by dynamically adjusting the prior provided by the pretrained model.
Outcome:	The proposed method improves monotonically on the mutual-information regularized RL objective.

Copied to clipboard

Xinjie Chen, Minpeng Liao, Guoxin Chen, Chengxi Li, Biao Fu, Kai Fan, Xinggao Liu

RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) has recently advanced the reasoning capabilities of large language models (LLMs).
Approach:	They propose a method that incorporates partial solution prefixes from expert demonstrations to guide the policy.
Outcome:	The proposed methods outperform strong baselines, yielding faster convergence and a higher performance ceiling.

Copied to clipboard

Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, Prathap Ramachandra

Feudal Reinforcement Learning for Dialogue Management in Large Domains (N18-2)

Challenge:	Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent.
Approach:	They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs.
Outcome:	The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment.

Copied to clipboard

Iñigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Stefan Ultes, Lina M. Rojas-Barahona, Bo-Hsiang Tseng, Milica Gašić

Natural Language Interfaces

Challenge:	Reinforcement learning (RL) is a promising approach to model dialogue policy optimisation but fails to scale to large domains due to the curse of dimensionality.
Approach:	They propose a novel approach to dialogue policy optimisation using reinforcement learning . they propose to decompose the decision into two steps using a domain ontology .
Outcome:	The proposed architecture outperforms state-of-the-art in several dialogue domains without any additional reward signal.

Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search (2026.findings-acl)

Copied to clipboard

Chuzhan Hao, Wenfeng Feng, Guochao Jiang, Guofeng Quan, Guohua Liu, Yuewei Zhang

Don’t Tell the Answer, Truly Guide the Reasoning During RL Rollouts (2026.findings-acl)

Challenge:	Existing RL-based search agents rely on stochastic exploration, leading to inefficient reasoning trajectories and unstable training.
Approach:	They propose a framework to enhance the performance and training stability of search agents by transforming raw reasoning trajectories into hierarchical experience knowledge.
Outcome:	The proposed framework exhibits strong cross-task and cross-algorithm generalizations on multiple complex agentic search and mathematical reasoning benchmarks.

Copied to clipboard

Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Ma Shuguang, Fei Yu, Yanghua Xiao

Crossing the Reward Bridge: Expanding Reinforcement Learning with Verifiable Rewards Across Diverse Domains (2026.acl-long)

Challenge:	Existing methods such as GRPO often break down when task difficulty exceeds the model’s capacity, resulting in sparse rewards and inefficient training.
Approach:	They propose to measure the compatibility between external guidance and a model's intrinsic policy by introducing an adaptive framework to enhance reasoning performance while explicitly preserving high Affinity.
Outcome:	The proposed framework outperforms baseline models while maintaining high Affinity.

Copied to clipboard

Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, Dong Yu

Challenge:	Reinforcement learning with verifiable rewards (RLVR) has been effective on structured tasks, but its reliance on simple, rule-based verifiers creates a bottleneck.
Approach:	They propose a framework that uses a generative verifier to provide soft, probabilistic rewards.
Outcome:	The proposed framework outperforms existing models up to 10x their size and can be scalable and effective.

Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023.findings-emnlp)

Copied to clipboard

Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang

TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning (2026.findings-acl)

Challenge:	Experimental results prove that language models can learn from human feedback better, irrespective of sequence length . emergence of length bias often induces the model to favor longer outputs .
Approach:	They propose to separate reward modeling from the influence of sequence length by using the Product-of-Experts technique.
Outcome:	The proposed approach shows that language models perform better regardless of sequence length . the main expert is focused on understanding human intents, while the biased expert targets the identification and capture of length bias.

Copied to clipboard

Jinyang Wu, Chonghua Liao, Mingkuan Feng, Shuai Zhang, Zhengqi Wen, Haoran Luo, Ling Yang, Huazhe Xu, Jianhua Tao

Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimization (2026.findings-acl)

Challenge:	Existing RL methods rely on unstructured self-sampling to fit scalar rewards, resulting in inefficient rollouts.
Approach:	They propose a structured template-guided RL framework that augments policy optimization with explicit template guidance.
Outcome:	Experiments show that TemplateRL outperforms GRPO and GRPI by 99% on AIME and 41% on AMC with superior stability on weak models and remarkable cross-domain generalization.

Copied to clipboard

Yu Li, Sizhe Tang, Tian Lan

RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2025.emnlp-main)

Challenge:	Existing approaches to reinforcement learning for Large Language Models treat trajectories as independent chains and ignore critical steps that may disproportionally impact reasoning outcome.
Approach:	They propose a framework that recovers latent correlated reward structure across seemingly independent trajectories by identifying and merging functionally similar steps/nodes.
Outcome:	The proposed framework recovers latent correlated reward structure across seemingly independent trajectories.

Copied to clipboard

Jiahui Li, Lin Li, Tai-Wei Chang, Kun Kuang, Long Chen, Jun Zhou, Cheng Yang

Bootstrapped Policy Learning for Task-oriented Dialogue through Goal Shaping (2024.emnlp-main)

Challenge:	Experimental results demonstrate the superiority of our approach to aligning large language models with human preferences.
Approach:	They propose a method that evaluates and assigns specific credit to each token using an off-the-shelf reward model.
Outcome:	The proposed method evaluates and assigns specific credit to each token using an off-the-shelf reward model.

Copied to clipboard

Yangyang Zhao, Ben Niu, Mehdi Dastani, Shihan Wang

CLHA: A Simple Yet Effective Contrastive Learning Framework for Human Alignment (2024.lrec-main)

Challenge:	Despite the promise of reinforcement learning, achieving seamless knowledge transitions in complex dialogue environments is difficult.
Approach:	They propose a Bootstrapped Policy Learning framework which adaptively tailors progressively challenging subgoal curriculum for each complex goal through goal shaping.
Outcome:	The proposed framework has shown to be effective across four publicly available datasets with different difficulty levels.

Copied to clipboard

Feiteng Fang, Liang Zhu, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang

Visually-Guided Policy Optimization for Multimodal Reasoning (2026.acl-long)

Challenge:	Large language models (LLMs) have attracted considerable attention from academic and industrial communities due to their outstanding performance in various natural language processing tasks.
Approach:	They propose a Contrastive Learning Framework for Human Alignment to evaluate the noise within the data and dynamically adjust the training process.
Outcome:	The proposed framework surpasses other algorithms in terms of reward model scores, automatic evaluations, and human assessments on the widely used dataset "Helpful and Harmless"

Copied to clipboard

Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu, Yong Wang, Yanlin Wang, Man Zhang, Xiangxiang Chu

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs (2026.findings-eacl)

Challenge:	Existing RLVRs lack visual faithfulness due to text-dominated reasoning . a novel framework to reinforce visual focus during policy optimization is proposed .
Approach:	They propose a framework to reinforce visual focus during policy optimization using visual attention compensation mechanism.
Outcome:	The proposed framework exhibits better visual activation and superior performance in multimodal reasoning and visual-dependent tasks.

Copied to clipboard

Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li

CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning (2022.naacl-main)

Challenge:	Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments.
Approach:	They propose a variant that operates on a turn-level MDP formulation, instead of the commonly used token-level one.
Outcome:	The proposed method is more robust than the widely used GRPO algorithm and more efficient than token-level MDPs.

Copied to clipboard

Siddharth Verma, Justin Fu, Sherry Yang, Sergey Levine

Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint (2024.findings-acl)

Challenge:	Existing approaches to training dialogue agents are supervised learning, but this is prohibitively expensive and time-consuming.
Approach:	They propose offline reinforcement learning methods that can be used to train dialogue agents . offline reinforcement learn methods can be combined with language models to yield realistic dialogue agents.
Outcome:	The proposed method can be combined with language models to produce realistic dialogue agents . the results show that the offline method can achieve the goal of the proposed system .

Copied to clipboard

Zhipeng Chen, Kun Zhou, Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, Ji-Rong Wen

DART: Disambiguation-Aware Reasoning for Video-guided Machine Translation (2026.acl-long)

Challenge:	Existing reinforcement learning methods do not provide fine-grained supervision for complex reasoning tasks.
Approach:	They propose a reinforcement learning method that incorporates a generative model as the reward model and a token-level supervision model for RL training.
Outcome:	Experiments on 8 tasks show the proposed method is effective .

Copied to clipboard

Boyu Guan, Chuang Han, Yang Zhao, Chengqing Zong

Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error (2026.acl-long)

Challenge:	Video-guided Machine Translation (VMT) uses short video clips to enhance translation quality, but many samples are text-sufficient.
Approach:	They propose a framework that integrates multimodal large language models’ multimodal reasoning into video-guided machine translation by using a pipeline for constructing training data based on multimodal relevance to translation.
Outcome:	The proposed framework improves multimodal information utilization in video-guided machine translation, yielding gains in translation quality and computational efficiency.

Copied to clipboard

Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Clive Bai, Saiyong Yang, Yunfang Wu

Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting (2025.findings-naacl)

Challenge:	Existing approaches to RLVR train LMs based on their own on-policy responses and are constrained by the initial capability of LM.
Approach:	They propose an approach that hints LMs with their self-made mistakes without external guidance.
Outcome:	The proposed approach outperforms the normal group relative policy optimization and requires no external guidance.

Copied to clipboard

Mohamed Salim Aissi, Clément Romac, Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, Olivier Sigaud, Laure Soulier, Nicolas Thome

Reinforcement Learning with Large Action Spaces for Neural Machine Translation (2022.coling-1)

Challenge:	Reinforcement learning (RL) is a promising approach for aligning large language models knowledge with sequential decision-making tasks.
Approach:	They propose to use a contrastive loss framework to analyze the sensitivity of LLMs to prompt formulations following RL training in a textual environment.
Outcome:	The proposed framework improves the model's robustness and generalization capabilities by minimizing the model’s internal representations and salient tokens.

Copied to clipboard

Asaf Yehudai, Leshem Choshen, Lior Fox, Omri Abend

Machine Translation

Challenge:	Recent work has argued that the gains produced by Reinforcement learning are mostly due to promoting tokens that have already received a fairly high probability in pre-training.
Approach:	They hypothesize that the large action space is a main obstacle to RL’s effectiveness in MT by reducing the size of the vocabulary without changing the vocabulary.
Outcome:	The proposed method improves by 1.5 BLEU points on average.

A Teacher-Student Framework for Maintainable Dialog Manager (D18-1)

Copied to clipboard

Weikang Wang, Jiajun Zhang, Han Zhang, Mei-Yuh Hwang, Chengqing Zong, Zhifei Li

Natural Language Interfaces

Challenge:	Reinforcement learning (RL) is an attractive solution for task-oriented dialog systems . but extending RL-based systems to handle new intents and slots requires a system redesign .
Approach:	They propose a teacher-student framework to extend RL-based dialog systems . they propose to specify constraints held in the new dialog manager .
Outcome:	The proposed framework makes no assumption about unsupported intents and slots, making it possible to improve RL-based systems incrementally.

Safer-Instruct: Aligning Language Models with Automated Preference Data (2024.naacl-long)

Copied to clipboard

Taiwei Shi, Kai Chen, Jieyu Zhao

Teaching LLMs to Plan, Not Just Solve: Plan Learning Boosts LLMs Generalization in Reasoning Tasks (2025.findings-emnlp)

Challenge:	annotating preference data by humans is resource-intensive and creativity-demanding . existing methods face limitations in data diversity and quality .
Approach:	They propose a pipeline for annotating large-scale preference data without human annotators.
Outcome:	The proposed pipeline outperforms models fine-tuned on human-annotated safety preference data while maintaining a competitive edge in downstream tasks.

Copied to clipboard

Tianlong Wang, Junzhe Chen, Weibin Liao, Xueting Han, Jing Bai

Direct Metric Optimization for Image Captioning through Reward-Weighted Augmented Data Utilization (2024.acl-long)

Challenge:	Existing methods for reinforcement learning (RL) on self-generated data are limited in many domains.
Approach:	a new framework combines plan-based search with Step-level Advantage Preference Optimization to optimize plan learning.
Outcome:	The proposed framework improves in-domain performance and out-of-domain benchmarks.

Copied to clipboard

Takumi Takada, Yuma Suzuki, Hiroki Takushima, Hayato Tanoue, Haruki Sato, Aiswariya Kumar, Hiroki Nishihara, Takayuki Hori, Kazuya Ueki

Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions (2025.findings-emnlp)

Challenge:	Recent large-scale vision language models (VLMs) lack continuity between learning objective and performance metrics.
Approach:	They propose a lightweight final-metric-optimizing training method that replaces the expensive exploration process in RL with an offline, diverse text data augmentation method.
Outcome:	The proposed method achieves comparable performance to state-of-the-art RL method while saving hundreds of times more model forwarding iterations and greater amounts of computation time.

Copied to clipboard

Yue Qiu, Yujan Ting, Pei Dong, Terrence Chen, Weijing Huang

Question Answering

Challenge:	Reinforcement learning (RL) for large language models typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling.
Approach:	They propose a mixed-data approach to training large language models with varying reward clarity . they combine Multiple-choice questions (MCQs) with OE questions for which they use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators.
Outcome:	The mixed-data approach improves medical question-answering performance across model scales.

Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue (2024.findings-emnlp)

Copied to clipboard

Huifang Du, Shuqin Li, Minghao Wu, Xuejing Feng, Yuan-Fang Li, Haofen Wang

WPO: Enhancing RLHF with Weighted Preference Optimization (2024.emnlp-main)

Challenge:	Existing RL methods focus on generation tasks while neglecting dialogue state tracking (DST) for understanding.
Approach:	They propose a method that integrates RL into both understanding and generation tasks by introducing step-by-step rewards throughout the token generation.
Outcome:	The proposed approach achieves state-of-the-art results on three widely used datasets.

Copied to clipboard

Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu

Enhancing Reinforcement Learning with Dense Rewards from Language Model Critic (2024.emnlp-main)

Challenge:	Off-policy preference optimization suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization.
Approach:	They propose a method to simulate on-policy learning with off-police preference data.
Outcome:	The proposed method outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 and MT-bench.

Copied to clipboard

Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, Lei Meng

Efficient (Soft) Q-Learning for Text Generation with Limited Good Data (2022.findings-emnlp)

Challenge:	Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences, but the sparsity of these signals can lead to inefficient and unstable learning.
Approach:	They propose a framework that utilizes the critique capability of Large Language Models to produce intermediate-step rewards during RL training.
Outcome:	The proposed framework improves sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation.

Copied to clipboard

Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, Zhiting Hu

Exploiting Tree Structure for Credit Assignment in Reinforcement Learning with Large Language Models (2026.findings-acl)

Challenge:	Maximum likelihood estimation (MLE) is the predominant method for training text generation models.
Approach:	They propose a new RL formulation for text generation from the soft Q-learning perspective using path consistency learning to combine the best of on-/off-policy updates and learn effectively from sparse reward.
Outcome:	The proposed approach outperforms MLE and previous RL methods in a wide range of tasks.

Copied to clipboard

Hieu Tran, Zonghai Yao, Hong yu

trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback (2023.emnlp-main)

Challenge:	Reinforcement learning has shown strong promise for strengthening reasoning ability of large language models, but sparse, delayed rewards make token-level credit assignment a central challenge.
Approach:	They propose a critic-free algorithm that rewards tokens that change the solution.
Outcome:	The proposed algorithm improves on in-distribution benchmarks and out-of-disttribution settings.

Copied to clipboard

Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, Louis Castricato

Improving NMT Models by Retrofitting Quality Estimators into Trainable Energy Loss (2025.coling-main)

Challenge:	Current RLHF paradigms rely on Proximal Policy Optimization (PPO), which quickly becomes a challenge to implement and scale up to large architectures.
Approach:	They propose an open-source framework for reinforcement learning from human feedback . it allows for offline fine-tuning of large language models .
Outcome:	The framework can be used to fine-tune models up to and exceeding 70 billion parameters.

Copied to clipboard

Gahyun Yoo, Jay Yoon Lee

Multilinguality

Challenge:	Reinforcement learning has shown great promise in aligning language models with human preferences in a variety of text generation tasks, including machine translation.
Approach:	They propose a method that employs quality estimators as trainable loss networks to backpropagate to the NMT model.
Outcome:	The proposed method outperforms strong baselines and proximal policy optimizations on English-to-Mongolian translation.

R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning (2025.findings-emnlp)

Copied to clipboard

Yuan Li, Qi Luo, Xiaonan Li, Bufan Li, Qinyuan Cheng, Bo Wang, Yining Zheng, Yuxin Wang, Zhangyue Yin, Xipeng Qiu

Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning (2024.emnlp-main)

Challenge:	RAG systems that integrate external knowledge with Large Language Models often become bottlenecks due to their limited parameters compared to LLMs and their inability to perform step-by-step reasoning.
Approach:	They propose a model that integrates external knowledge with Large Language Models to enhance factual correctness and mitigate hallucination.
Outcome:	The proposed model outperforms baselines and can transfer well to different retrievers.

Copied to clipboard

Jiahui Li, Hanlin Zhang, Fengda Zhang, Tai-Wei Chang, Kun Kuang, Long Chen, Jun Zhou

TreeRL: LLM Reinforcement Learning with On-Policy Tree Search (2025.acl-long)

Challenge:	Recent research has developed algorithms for reinforcement learning from human feedback and AI-generated feedback.
Approach:	They propose a method for reinforcement learning from human feedback and AI-generated feedback that incorporates weighting, ranking, and constraining to handle disparate rewards.
Outcome:	The proposed method reduces disparity and enhances stability among rewards . empirical results show that the proposed method is efficient and straightforward .

Copied to clipboard

Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, Yuxiao Dong

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)

Challenge:	Existing methods for On-Policy LLM RL typically train a separate process reward model, which suffers from distribution mismatch and reward hacking.
Approach:	They propose a reinforcement learning framework that directly incorporates on-policy tree search for RL training.
Outcome:	Experiments on math and code reasoning benchmarks show that tree search achieves superior performance compared to traditional ChainRL.

Copied to clipboard

Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang

Explainability & Interpretability in NLP

Challenge:	Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences.
Approach:	They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data.
Outcome:	The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark .

Bridging Reasoning and Action: Hybrid LLM–RL Framework for Efficient Cross-Domain Task-Oriented Dialogue (2026.findings-acl)

Copied to clipboard

Yangyang Zhao, Linfan Dai, Li Cai, Bowen Xing, Libo Qin

DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing (2026.acl-long)

Challenge:	Existing methods to solve cross-domain task-oriented dialogues are brittle when cross- domain constraints are not directly grounded in surface text or require commonsense inference.
Approach:	They propose a framework that makes LLM-derived constraint reasoning usable for RL.
Outcome:	Experiments show that the proposed framework outperforms single-model baselines on long-horizon tasks.

Copied to clipboard

Qian Cao, Yahui Liu, Wei Bi, Yi Zhao, Ruihua Song, Xiting Wang, Ruiming Tang, Guorui Zhou, Han Li

Challenge:	Existing methods for enhancing large language models (LLMs) lack explicit mechanisms for guiding diverse exploration and instead prioritize efficiency and performance over diversity.
Approach:	They propose a reinforcement learning-based framework that decomposes the generation process into explicitly planned intermediate steps and introduces divergence at the planning phase based on diversity variation.
Outcome:	The proposed method significantly outperforms existing baselines on creative writing benchmarks on a semi-structured long chain-of-thought (CoT) it introduces divergence at the planning phase based on diversity variation, alongside a group-aware diversity reward to encourage distinct trajectories.

Agentic Episodic Control (2026.findings-acl)

Copied to clipboard

Xidong Yang, Wenhao Li, Junjie Sheng, Yun Hua, Haosheng Chen, Chuyun Shen, Xiangfeng Wang

Green & Sustainable NLP

Challenge:	Existing methods for reinforcement learning (RL) are limited by poor data efficiency and weak generalization.
Approach:	They propose a novel architecture that integrates large language models into episodic RL.
Outcome:	The proposed architecture achieves 2–6 higher data efficiency than baselines and is the only method to solve complex tasks like UnlockLocal with over 90% success.

Graph-Reward-SQL: Execution-Free Reinforcement Learning for Text-to-SQL via Graph Matching and Stepwise Reward (2025.findings-emnlp)

Copied to clipboard

Han Weng, Puzhen Wu, Cui Longjie, Yi Zhan, Boyi Liu, Yuanfeng Song, Dun Zeng, Yingxiang Yang, Qianru Zhang, Dong Huang, Xiaoming Yin, Yang Sun, Xing Chen

A Collaborative Reasoning Framework Powered by Reinforcement Learning and Large Language Models for Complex Questions Answering over Knowledge Graph (2025.coling-main)

Challenge:	Existing methods to enhance performance of large language models (LLMs) on Text-to-SQL tasks rely on execution-based or LLM-based reward models.
Approach:	They propose a reward model framework for RL-based Text-to-SQL that employs the GMNScore outcome reward model.
Outcome:	The proposed reward model outperforms existing reward models on standard benchmarks including Spider and BIRD.

Copied to clipboard

Zhiqiang Zhang, Wen Zhao

PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning (2026.findings-acl)

Challenge:	Knowledge Graph Question Answering (KGQA) aims to answer natural language questions by reasoning across multiple triples in knowledge graphs.
Approach:	They propose a collaborative reasoning framework powered by RL and LLMs to answer complex questions based on the knowledge graph.
Outcome:	The proposed model surpasses state-of-the-art models on four datasets.

Copied to clipboard

Yunzhi Shen, Hao Zhou, Xin Huang, Xue Han, Junlan Feng, Shujian Huang

SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models (2026.findings-acl)

Challenge:	Reinforcement learning (RL) has shown strong promise for LLM-based machine translation . however, translation-oriented RL remains challenged by high-variance policy gradients induced by Monte Carlo baselines and large trajectory space that favors global exploration over fine-grained local optimization.
Approach:	They propose a two-stage RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization.
Outcome:	The proposed framework supports global exploration and fine-grained optimization while supporting global exploration.

Copied to clipboard

Yifu Huo, Chenglong Wang, Ziming Zhu, Shunjie Xing, Peinan Feng, Tongran Liu, Qiaozhi He, Tian Hua Zhou, null Changxiaojia, JingBo Zhu, Zhengtao Yu, Tong Xiao

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging (2024.emnlp-main)

Challenge:	Reinforcement learning (RL) training typically improves single-sample success rates but limited exploration of diverse reasoning trajectories.
Approach:	They propose a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL) they propose 'Steering Probability Squeezing' to enhance exploration without external supervision .
Outcome:	The proposed training paradigm improves Pass@k and improves exploration of diverse reasoning trajectories without external supervision.

Copied to clipboard

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

GAPO: Robust Advantage Estimation for Real-World Code LLMs (2026.findings-acl)

Challenge:	Modern large language models (LLMs) showcase impressive capabilities across various tasks with aligning their behavior with human preferences.
Approach:	They propose a framework that integrates domain-specific knowledge into a general reward model by model merging.
Outcome:	The proposed framework improves performance across different benchmarks and provides detailed analysis showing the effects of model merging.

Copied to clipboard

Jianqing Zhang, Zhezheng Hao, Wei Xia, Hande Dong, Hong Wang, Chenxing Wei, Yuyan Zhou, Yubin Qi, Qiang Lin, Jian Cao

Robustness in NLP

Challenge:	Reinforcement learning (RL) is widely used for post-training large language models (LLMs) in code editing, but in real-world code editing scenarios, reward distributions are often skewed with unpredictable noise, leading to distorted advantage computation and increased rollout outliers.
Approach:	They propose a group-relative method that finds an interval with the highest SNR and uses the median of that interval as an adaptive Q to replace the group mean in advantage calculation.
Outcome:	The proposed method improves on nine instruction-tuned LLMs while remaining plug-and-play and efficient.

Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization (2026.findings-acl)

Copied to clipboard

Huimin Xu, Shuai Zhao, Xiaobao Wu, Anh Tuan Luu

Triviality Corrected Endogenous Reward (2026.acl-long)

Challenge:	Existing RLVR algorithms suffer from entropy collapse, leading to premature determinism and unstable optimization.
Approach:	They propose an adaptive entropy flow balancing mechanism that rescales entropic-increasing and enotro-decreazing updates according to their contributions to enthroy change.
Outcome:	The proposed method outperforms existing RLVR algorithms on six reasoning benchmarks.

Copied to clipboard

Xinda Wang, Zhengxu Hou, Yangshijie Zhang, null Yanbingren, Jialin Liu, ChenZhuo Zhao, Zhibo Yang, Bin-Bin Yang, Feng Xiao

Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay (2026.findings-acl)

Challenge:	Recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards focuses on open-ended text generation, requiring either annotated data or powerful closed-source models.
Approach:	They propose a method that rewards the relative information gain between a specialist and a generalist reference policy, modulated by a probability-dependent correction mechanism.
Outcome:	The proposed model improves on multiple writing benchmarks and model architectures without external supervision and validates generality across different generation tasks.

Copied to clipboard

Jianqing Zhang, Wei Xia, Zhezheng Hao, Hong Wang, Hande Dong, Qiang Lin, Yang Liu, Jian Cao, Qiang Yang

Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures (2026.acl-long)

Challenge:	Existing approaches to reinforcement learning (RL) rely on static, in-epoch metrics that overlook training dynamics, often introducing low-utility or outdated data.
Approach:	They propose a plug-and-play module that prioritizes cross-epoch ambiguous samples to neutralize the noise from stale experiences.
Outcome:	Extensive experiments on nine LLMs show that Adaptive Ambiguity Replay outperforms state-of-the-art baselines on real-world code editing tasks.

Copied to clipboard

Yi Hu, Jiaqi Gu, Ruxin Wang, Zijun Yao, Hao Peng, Xiaobao Wu, Jianhui Chen, Muhan Zhang, Liangming Pan

RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models (2026.findings-acl)

Challenge:	Recent research has shown that reinforcement learning can elicit intriguing emergent reasoning behaviors.
Approach:	They propose a comprehensive survey of the mechanistic understanding of large reasoning models . they organize findings into three core dimensions: 1) training dynamics, 2) reasoning mechanisms, and 3) unintended behaviors.
Outcome:	This paper synthesizes the mechanistic understanding of large reasoning models into three dimensions . authors outline a roadmap for future studies including improved interpretability and methodologies .

Copied to clipboard

Tianqianjin Lin, Xi Zhao, Xingyao Zhang, Rujiao Long, Yi Xu, Zhuoren Jiang, Wenbo Su, Bo Zheng

Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models (2024.findings-emnlp)

Challenge:	Experiments show that reinforcement learning (RL) can refine the reasoning abilities of large language models (LLMs) but requires a key prerequisite: the model must already be able to generate high-utility reasoning paths with non-negligible probability.
Approach:	They propose a framework that uses answer-conditioned reasoning as a variational surrogate for question-only reasoning.
Outcome:	Experiments on 11 benchmarks and 3 models show that RAVR reduces hesitation, strengthens conclusion consolidation, and promotes problem-specific strategies in reasoning.

Copied to clipboard

Muhan Lin, Shuyang Shi, Yue Guo, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Simon Stepputtis, Joseph Campbell, Katia Sycara

Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics (2025.findings-emnlp)

Challenge:	Reward hacking is a problem in reinforcement learning where the ability to specify the desired behavior of a reward function is difficult.
Approach:	They propose to use feedback as a potential-based shaping function to solicit and apply feedback from large language models to improve convergence speed and policy returns.
Outcome:	The proposed method improves convergence speed and policy returns over baselines even with significant ranking errors and eliminates the need for complex post-processing of reward functions.

Copied to clipboard

Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson

Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards (2026.acl-long)

Challenge:	Existing evaluation metrics are unreliable for factual consistency tasks, limiting their effectiveness as signals for shaping model behaviour.
Approach:	They propose an automated training pipeline that improves factual consistency in summaries by aggregating scores from different weak metrics.
Outcome:	The proposed approach improves factual consistency in summaries by aggregating scores from weak metrics.

Copied to clipboard

Jiajie Zhang, Xin Lv, Ling Feng, Lei Hou, Juanzi Li

IPO: Your Language Model is Secretly a Preference Classifier (2025.acl-long)

Challenge:	Existing methods for reinforcement learning (RL) rely on binary outcome rewards that fail to capture the comprehensiveness and factuality of agents’ reasoning process.
Approach:	They propose a reward framework that emphasizes reasoning comprehensiveness, factual grounding, and evidence connectivity.
Outcome:	The proposed framework outperforms standard outcome-based RL baselines across multiple deep search benchmarks and shows that it discourages shortcut exploitation and promotes comprehensive, evidence-grounded reasoning.

Copied to clipboard

Shivank Garg, Ayush Singh, Shweta Singh, Paras Chopra

Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning (2026.acl-long)

Challenge:	Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences, but it often incurs significant computational and financial costs due to its reliance on training external reward models or human-labeled preferences.
Approach:	They propose an alternative approach that leverages generative LLMs as preference classifiers to reduce the dependence on external reward models or human-labeled preferences.
Outcome:	The proposed approach reduces the dependence on external reward models or human-labeled preferences by using generative LLMs as preference classifiers.

Copied to clipboard

Yangyi Fang, Haolin Shi

d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models (2026.acl-long)

Challenge:	Reinforcement learning (RL) training on easy problems can cause overfitting and pass@k degradation, while training on hard problems yields sparse reward signals.
Approach:	They propose a hint injection framework that strategically identifies and provides critical reasoning steps during training.
Outcome:	Experiments on six mathematical reasoning benchmarks show that the proposed framework achieves comparable average performance to 32B baselines while preserving pass@k diversity across all k values.

Copied to clipboard

Leyi Pan, Shuchang Tao, Yunpeng Zhai, Zheyu Fu, Liancheng Fang, Minghua He, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, Lijie Wen

EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning for LLMs (2026.findings-acl)

Challenge:	Existing RL methods suffer from reliability bottlenecks due to reward sparsity and intractable computations . d-TreeRPO provides fine-grained and verifiable step-wise reward signals .
Approach:	They propose a reliable reinforcement learning framework for diffusion large language models that leverages tree-structured rollouts and bottom-up advantage computation based on verifiable outcome rewards.
Outcome:	The proposed framework outperforms baseline models and achieves significant improvements across reasoning benchmarks.

Copied to clipboard

Huanyu Liu, Jia Li, Yihong Dong, Chang Yu, Taozhi Chen, Lecheng Wang, Yongding Tao, Bin Gu, Ge Li

Breaking Consensus Bias: Unsupervised Reinforcement Learning for Machine Translation (2026.findings-acl)

Challenge:	Existing approaches to reinforcement learning with verifiable reward (RLVR) are limited by difficulty or lack of exploration.
Approach:	They propose a self-evolving curriculum learning framework based on chain-of-thought reasoning optimization that constrains exploration space by self-generating and verifying CoT trajectories.
Outcome:	The proposed framework enables LLMs to solve previously unsolved problems without external supervision and is compatible with various RL fine-tuning methods.

Copied to clipboard

Shuting Jiang, Ran Song, Siqi Zhang, Yuxin Huang, Shengxiang Gao, Zhengtao Yu

Machine Translation

Challenge:	Existing RL approaches for MT face fixed references or the production of homogeneous references leading to mode collapse in unsupervised settings.
Approach:	They propose an Entropy-Driven Unsupervised RL framework for machine translation that leverages entropy for supervision construction and self-evolution.
Outcome:	The proposed framework outperforms supervised and unsupervised baselines in multiple language pairs.

A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions (2026.acl-long)

Copied to clipboard

Zhiyin Yu, Yuchen Mou, Juncheng Yan, Junyu Luo, Chunchun Chen, Xing Wei, Yunhui Liu, Hongru Sun, Yuxing Zhang, Jun Xu, Yatao Bian, Ming Zhang, Wei Ye, Tieke He, Jie Yang, Guanjie Zheng, Zhonghai Wu, Bo Zhang, Lei Bai, Xiao Luo

CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution (2026.acl-long)

Challenge:	Existing research on reinforcement learning for LLMs under data scarcity has not been unified.
Approach:	They propose a top-up hierarchical framework built around three complementary perspectives: data-centric, training-centric and framework-centric.
Outcome:	The proposed framework provides a clear conceptual foundation for understanding the design space of data-efficient RL for large language models and to guide researchers working in this emerging area.

Copied to clipboard

Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang, Xiangxiang Chu

Challenge:	Extensive experiments on AppWorld and BFCL demonstrate consistent and significant improvements over strong base models, yielding absolute gains of 19.43%, 15.58%, and 18.14%, respectively.
Approach:	They propose a framework that extracts feedback signals such as forgetting and uncertainty from rollout trajectories and utilizes them to guide LLM-based task synthesis.
Outcome:	Extensive experiments on AppWorld and BFCL show that the proposed framework improves over strong base models.

I²B-LPO: Latent Policy Optimization via Iterative Information Bottleneck (2026.acl-long)

Copied to clipboard

Huilin Deng, Hongchen Luo, Yue Zhu, Long Li, Zhuoyue Chen, Xinghao Zhao, Ming LI, Chuyang Zhao, Jihai Zhang, MengChang Wang, Yang Cao, Yu Kang

SPARK: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning (2026.acl-long)

Challenge:	Existing methods for large language model reasoning suffer from exploration collapse due to the semantic homogeneity of random rollouts.
Approach:	They propose to use latent policy optimization via iterative information bottleneck to optimize reasoning trajectories by diversifying reasoning .
Outcome:	Empirical results show that the proposed method achieves state-of-the-art performance with margins of up to 5.3% in accuracy and 7.4% in diversity metrics.

Copied to clipboard

Jinyang Wu, Shuo Yang, Yuhao Shen, Shuai Zhang, Zhengqi Wen, Jianhua Tao

Challenge:	Existing methods for training large language models waste computation budget on trivial steps while failing to guarantee sample quality.
Approach:	They propose a framework that selectively branches at critical decision states for resource-efficient exploration.
Outcome:	The proposed framework activates adaptive branching exploration at critical decision states to probe promising trajectories, thereby achieving precise resource allocation that prioritizes sampling quality over blind coverage.

AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin (2026.acl-long)

Copied to clipboard

Jian Xiong, Jingbo Zhou, Jingyong Ye, Qiang Huang, Dejing Dou

GR1: Reinforcement-Enhanced LLM for Geoscience Reasoning (2026.findings-acl)

Challenge:	Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models.
Approach:	They propose an algorithm that optimizes cross-entropy loss using advantages enhanced through a margin-based estimation scheme.
Outcome:	Experimental results show that AAPO improves group relative advantage estimation compared to other methods.

Copied to clipboard

Yule Xie, Jiaxin Ding, Cheng Deng, Shiqing Gao, Junran Zhang, Sibo Zhang, Zeyuan Wang, Ke Wu, Xin Ding, Luoyi Fu, Meng Jin, Xinbing Wang

Curiosity-Driven Reinforcement Learning from Human Feedback (2025.acl-long)

Challenge:	Recent advances in large language models have demonstrated RL's substantial capacity to enhance multi-step reasoning beyond what supervised instruction tuning achieves.
Approach:	They propose a framework that converts multimodal questions into descriptive text . they propose RL-enhanced geoscience reasoning that can be fine-tuned to a text-only level .
Outcome:	The proposed framework improves accuracy and accuracy on multimodal questions while preserving answerability and difficulty.

Copied to clipboard

Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang

Challenge:	Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models with human preferences, but often at the cost of reduced output diversity.
Approach:	They propose a framework that incorporates intrinsic rewards for novel states alongside traditional sparse extrinsic rewards to optimize both output diversity and alignment quality.
Outcome:	The proposed framework achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF.

Progressive Planning and Reinforced Reasoning: Large Language Model-Guided Multi-hop Question Answering over Knowledge Graph (2026.findings-acl)

Copied to clipboard

Xiang Li, Runhai Jiao, Ruifan Li, Dongnan Wu, Ruojiao Qiao, Lei Liu

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts (2026.acl-long)

Challenge:	Existing approaches to multi-hop question answering lack effective intermediate guidance and policy networks focus on local neighborhood information, making it difficult to anticipate the long-term consequences of decisions.
Approach:	They propose a framework that converts decomposed sub-question sequences into stepwise decision guidance and a structure-aware lookahead policy network to enhance the agent's global state awareness and decision foresight in complex environments.
Outcome:	The proposed framework surpasses state-of-the-art methods while showing strong generalization.

Copied to clipboard

Di Zhang, Xun Wu, Shaohan Huang, Lingjie Jiang, Yaru Hao, Li Dong, Zewen Chi, Zhifang Sui, Furu Wei

TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles (2026.acl-long)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) training with Mixture-of-Experts policies remains fragile and prone to reward collapse.
Approach:	They propose a router shift-based policy optimization method that computes a per-token router-shift ratio conditioned on the previously activated experts and applies stop-gradient and a lower-bound floor.
Outcome:	The proposed method achieves better performance and greater stability than previous methods.

Copied to clipboard

Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu

Challenge:	Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints.
Approach:	They propose a framework that uses tiny language models to evaluate instruction following . they propose to use a set of specialized tiny language model to provide rewards for soft constraints.
Outcome:	The proposed framework outperforms baseline models by 12% and speeds up training time by 3.

Revisiting Entropy in Reinforcement Learning for Large Reasoning Models (2026.findings-acl)

Copied to clipboard

Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, Deyi Xiong

Filtered Direct Preference Optimization (2024.emnlp-main)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) has emerged as a paradigm for enhancing the reasoning capabilities of large language models.
Approach:	They propose a positive-advantage reweighting approach that regulates model entropy by adjusting the loss weights assigned to tokens with positive advantages during RLVR training.
Outcome:	The proposed approach regulates model entropy by adjusting loss weights assigned to tokens with positive advantages during RLVR training while maintaining competitive performance.

Copied to clipboard

Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, Kaito Ariu

Challenge:	Existing studies on the impact of RLHF on text quality have focused on reward-model-free RL.
Approach:	They propose an extension of direct preference optimization to improve model performance by analyzing the quality of the preference dataset.
Outcome:	The proposed method improves the performance of models optimized with DPO over those optimized with reward-model-based RLHF.

Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening (2025.emnlp-main)

Copied to clipboard

Andre Wang He, Daniel Fried, Sean Welleck

CPC-GRPO: Answer-Free Reinforcement Learning with Cross-Prompt Consensus Rewards (2026.findings-acl)

Challenge:	Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities.
Approach:	They propose a method for explicitly up-weighting rare but correct solutions to overcome rank bias in group relative policy optimization (GRPO) .
Outcome:	The proposed method mitigates rank bias and improves pass@N across a large range of N in both synthetic and real theorem proving settings.

Copied to clipboard

Gyunyeop Kim, Sangwoo Kang

Behavior Knowledge Merge in Reinforced Agentic Models (2026.acl-long)

Challenge:	Reinforcement learning with verifiable rewards is a popular post-training tool for large language models, but relies on a ground-truth answer or external verifier, which limits applicability and increases cost.
Approach:	They propose an answer-free training objective that derives rewards solely from the model’s own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent.
Outcome:	The proposed objective derives rewards solely from the model’s own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent.

Copied to clipboard

Xiangchi Yuan, Dachuan Shi, Chunhui Zhang, Zheyuan Liu, Shenglong Yao, Soroush Vosoughi, Wenke Lee

Challenge:	Existing methods for supervised fine-tuning (SFT) are suboptimal to preserve task-specific capabilities on RL-trained agentic models.
Approach:	They propose a distribution-aware merging framework specifically designed for RL-trained agentic models that disentangles shared and task-specific unique parameter updates while selectively preserving and rescaling unique ones.
Outcome:	Experiments across multiple agent domains and model architectures show that the proposed framework surpasses baselines and unlocks synergistic potential among agents.

Tailored Primitive Initialization is the Secret Key to Reinforcement Learning (2026.acl-long)

Copied to clipboard

Yihang Yao, Guangtao Zeng, Raina Wu, Yang Zhang, Ding Zhao, Zhang-Wei Hong, Chuang Gan

MARS2: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation (2026.acl-long)

Challenge:	Reinforcement learning (RL) has emerged as a powerful paradigm for improving the reasoning capabilities of large language models.
Approach:	They propose a pipeline that automatically discovers thinking token patterns with reasoning primitives and curates SFT datasets to prepare LLMs for RL.
Outcome:	The proposed pipeline outperforms baseline methods on mathematical and logical reasoning benchmarks on RL tasks.

Copied to clipboard

Pengfei Li, Shijie Wang, Fangyuan Li, Yikun Fu, Kaifeng Liu, Kaiyan Zhang, Dazhi Zhang, Yuqiang Li, Biqing Qi, Bowen Zhou

Programming Languages in NLP

Challenge:	Existing approaches to reinforcement learning are decoupled from structured search due to limited trajectory diversity.
Approach:	They propose a unified RL framework that integrates multiple agents within a shared tree-structured search environment.
Outcome:	Experiments show that MARS2 improves performance across diverse model combinations and training settings.

View-R1: Asymmetric Policy Optimization for Difficulty-Aware Multimodal Reinforcement Learning (2026.findings-acl)

Copied to clipboard

Minjie Hong, Zirun Guo, Jiabao Zhang, Zehan Wang, Ziang Zhang, Tao Jin, Zhou Zhao

VerIF: Verification Engineering for Reinforcement Learning in Instruction Following (2025.emnlp-main)

Challenge:	Multimodal Large Language Models (MLLMs) are powerful at integrating diverse data but struggle with complex reasoning.
Approach:	They propose a method which separates responses into positive and negative groups to stabilize training and preserve knowledge.
Outcome:	The proposed model View-R1 achieves a 10.55% improvement in reasoning and outperforms larger models while maintaining and improving performance on general tasks.

Copied to clipboard

Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li

Challenge:	Best practices for RL in instruction following remain underexplored.
Approach:	They propose a verification method that combines rule-based code verification with LLM-based verification from a large reasoning model.
Outcome:	The proposed method achieves state-of-the-art performance among models of comparable size and generalizes well to unseen constraints.

SCALER: Synthetic Scalable Adaptive Learning Environment for Reasoning (2026.findings-acl)

Copied to clipboard

Caijun Xu, Changyi Xiao, Zhongyuan Peng, Xinrun Wang, Yixin Cao

Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation (2026.findings-acl)

Challenge:	Reinforcement learning (RL) is a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve.
Approach:	They propose a framework that sustains effective learning signals through adaptive environment design that transforms real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation.
Outcome:	The proposed framework outperforms baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics.

Copied to clipboard

Yifeng Liu, Siqi Ouyang, Yatish H R, Lei Li

Multilinguality

Challenge:	Existing methods for training large language models rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages.
Approach:	They propose a reinforcement training method using only monolingual text to elevate LLMs’ translation capabilities on massive low-resource languages while retaining their performance on high-resourced languages.
Outcome:	The proposed model outperforms LLaMAX, one of the strongest open-source multilingual LLMs on 1,414 language directions on Flores-101 dataset.

Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models (2026.findings-acl)

Copied to clipboard

Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, Deren Lei

AG-GRPO: Answer-Guided GRPO for Masked Diffusion Language Models (2026.acl-long)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning . however, the recipe introduces a significant risk of capability regression, where models forget foundational skills after prolonged training without employing regularization strategies.
Approach:	They propose a replay strategy with dynamic objective reweighting for general knowledge preservation using short-horizon signals of convergence and instability.
Outcome:	The proposed method preserves general capabilities and improves reasoning . it can be applied to existing RLVR pipelines without training additional models or tuning .

Copied to clipboard

Juhyeong Kim, Gyunyeop Kim, Sangwoo Kang

CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning (2026.acl-long)

Challenge:	Recent work on large language models (LLMs) has emphasized not only final-answer accuracy but also reliability of reasoning on challenging tasks.
Approach:	They propose an answer-guided group-relative policy optimization for masked diffusion language models which generates text through iterative mangled token restoration.
Outcome:	The proposed approach improves over pretrained dLLMs and prior RL methods across mathematics, puzzle-solving, and code-generation benchmarks.

Copied to clipboard

Zhenpeng Su, Leiyu Pan, Minxuan Lv, Yuntao Li, Wenping Hu, Fuzheng Zhang, Kun Gai, Guorui Zhou

StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason (2026.acl-long)

Challenge:	Existing methods for proximal policy optimization discard valuable gradient signals from low-probability tokens due to the clipping mechanism.
Approach:	They propose an algorithm that reintroduces gradients from clipped tokens in native PPO in a gentle and bounded manner.
Outcome:	The proposed algorithm outperforms strong baselines on reasoning benchmarks on different model scales.

Copied to clipboard

Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan hu, Rui Yan

Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning (2026.findings-acl)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) approaches face two challenges: the near-miss reward problem and exploration stagnation.
Approach:	They propose an algorithm that partitions valid reasoning chains into reasoning steps using multi-level stepwise hints.
Outcome:	The proposed method outperforms competing RLVR enhancement methods across six mathematical benchmarks and two out-of-domain benchmarks.

Copied to clipboard

Haolin Liu, Dian Yu, Sidi Lu, Yujun Zhou, Rui Liu, Zhenwen Liang, Haitao Mi, Chen-Yu Wei, Dong Yu

Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models (2026.acl-long)

Challenge:	Existing reinforcement learning methods rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.
Approach:	They propose a process reward model that rewards correct steps only when they detect errors . they propose VPPO, which rewards the correct prefix and an erroneous suffix .
Outcome:	a new approach outperforms sparse-reward RL and prior PRM-guided baselines on Pass@1 and Pass@K . a process reward model (PRM) outperformed sparser-rebound RL on multiple reasoning benchmarks .

Copied to clipboard

Dan Shi, Zhuowen Han, Simon Ostermann, Renren Jin, Josef Van Genabith, Deyi Xiong

On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning (2026.acl-long)

Challenge:	Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting.
Approach:	They propose a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup.
Outcome:	The proposed method identifies a compact, task-agnostic set of features that directly mediate generalization across diverse tasks.

Copied to clipboard

Di Wu, Jiaxin Fan, Chloe Gu, Guanbo Wang, Wei Yin, Wenhao Li, Bo Jin

MTP-RL: Acceleration of Reinforcement Learning Rollouts with Policy-Aligned Multi-Token Prediction (2026.findings-acl)

Challenge:	Embodied planning requires agents to make coherent multi-step decisions based on dynamic visual observations and verbal goals.
Approach:	They propose an On-policy Reinforcement fine-tuning framework with offline rewards for Embodied Task Planning that preserves generalization benefits of RFT while addressing costly interaction and sparse rewards.
Outcome:	The proposed framework outperforms closed-source and online-RL methods on EmbodiedBench, a recent benchmark for interactive embodied tasks.

Copied to clipboard

Ke Wang, Aohan Zeng, Zhengxiao Du, Yuxuan Hu, Bohan Zhang, Xinyi Wang, Jie Tang, Jing Zhang

Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains (2026.acl-long)

Challenge:	Reinforcement learning (RL) is widely applied to boost the performance of pretrained models, yet its training efficiency is severely constrained by rollout generation.
Approach:	They propose a framework that accelerates the rollout phase for diverse models by equipping a pipeline to equip the multi-layer parameter-sharing MTP for all models and an advantage-aware MTP optimization strategy.
Outcome:	The proposed framework achieves stable growth of acceptance length during RL training, and also accelerates RL rollouts, achieving an average 23.1%–55.3% reduction in rollout time compared to baselines.

Copied to clipboard

Zhonghang Yuan, Zhefan Wang, Fang Hu, Zihong Chen, Jinzhe Li, Gang Li, Jie Ying, Huanjun Kong, Songyang Zhang, Nanqing Dong

Improved Policy Optimization for Mixture-of-Experts Models: Importance Sampling and Rewarding from an Expert-Centric Perspective (2026.findings-acl)

Challenge:	Recent large language models (LLMs) have demonstrated remarkable progress in reasoning, but their applications on knowledge-intensive domains have not been explored due to the scarcity of high-quality verifiable data.
Approach:	They propose a framework that extends reinforcement learning with verifiable rewards (RLVR) to knowledge-intensive domains through automated verififiability data synthesis while enabling verification of the LLM's reasoning process.
Outcome:	Extensive experiments show that the proposed framework enhances the reasoning of large language models in knowledge-intensive domains without significantly compromising the model’s general capabilities.

Copied to clipboard

Yining Qian, Jinpeng Li, Fei Mi, Lifeng Shang, Xiang Zhang

Challenge:	Existing approaches to reinforcement learning (RL) suffer from training instability . existing approaches often ignore token-specific discrepancies in expert assignments .
Approach:	They propose to introduce expert-level importance sampling to reduce complexity of RL . they propose to leverage expert-centric granularity to ensure a rigorous alignment between reward signals and policy updates.
Outcome:	The proposed method outperforms strong baselines across reasoning tasks.

Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards (2026.acl-long)

Copied to clipboard

Shaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong, Guangyi Chen, Eric P. Xing, Kun Zhang

MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following (2026.acl-long)

Challenge:	Existing methods for improving reasoning in diffusion language models rely on outcome-based rewards that provide no direct supervision over the denoising process.
Approach:	They propose a method that provides a process-level reinforcement signal over denoising trajectory of diffusion language models.
Outcome:	Experiments on challenging reasoning benchmarks show that the proposed model improves reasoning stability, interpretability and overall performance.

Copied to clipboard

Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili, Mohammad Javad Dousti

Challenge:	Large language models (LLMs) can follow many natural-language instructions, yet they remain brittle when a request bundles multiple explicit constraints, such as asking the LLM to respond in a particular structure with an exact ending phrase.
Approach:	They propose a method which stabilizes learning through multi-temperature sampling to increase reward dispersion, dual-anchor advantages to restore gradients in homogeneous groups, prospect-theoretic shaping to bound updates and penalize violations based on Kahneman Tversky’s theory and asymmetric KL regularization.
Outcome:	The proposed method outperforms standard GRPO on FollowBench, IFEval, and a curated multi-constraint dataset, improving strict constraint satisfaction by up to 5.0% on Llama-3.2-3B.

Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs (2026.findings-acl)

Copied to clipboard

Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi

Information Extraction & Text Mining

Challenge:	Reinforcement learning (RL) is a paradigm for post-training large language models, but it suffers from exploration collapse . a new study finds that RL fails to reward correct solutions that exhibit rare high-level strategies .
Approach:	They propose a method that rewards correct solutions that exhibit rare high-level strategies by clustering rollouts according to their high- level solution strategies.
Outcome:	The proposed approach improves pass@k across large sampling budgets and increases area under the pass@K curve (AUC@K) without sacrificing pass@1.

Entropy-Aware Reshaping of Reinforcement Signals for Multi-Answer Reasoning (2026.findings-acl)

Copied to clipboard

Zhi Li, Huidan Xu, Zhen Hu, Yali Du, Ying Liu

LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment (2026.findings-acl)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) is a standard post-training paradigm for large language models.
Approach:	They propose a framework that reshapes how learning signals are normalized and aggregated.
Outcome:	Experiments on MCTACO and MMLU-Multi show that the proposed framework improves accuracy, training stability and cross-dataset transfer performance.

Copied to clipboard

Shipeng Li, Zhiqin Yang, Shikun Li, Xiaobo Xia, Hengyu Liu, Xinghua Zhang, Gaode Chen, Dong Fang, Ying Tai, Zhe Peng

LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance (2026.acl-long)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) is a key technique for enhancing LLMs’ reasoning abilities, yet its data inefficiency remains a major bottleneck.
Approach:	They propose a gradient-alignment-based method which intelligently selects the learnable and representative training reasoning data for RLVR post-training.
Outcome:	Experiments on five reasoning benchmarks show that the proposed method significantly reduces training data requirements while improving performance.

Copied to clipboard

Yuchun Fan, Bei Li, Peiguang Li, Yilin Wang, Yongyu Mu, Jian Yang, Xin Chen, Rongxiang Weng, Jingang Wang, Xunliang Cai, JingBo Zhu, Tong Xiao

Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback (2026.findings-acl)

Challenge:	Existing methods for enhancing multi-step reasoning have not fully translated to multilingual contexts.
Approach:	They propose a framework that leverages language-conditioned hints to guide exploration in non-English reasoning tasks.
Outcome:	Empirical results show that the proposed framework improves reasoning performance without compromising language consistency.

Copied to clipboard

Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe

Speech & Audio in NLP

Challenge:	Existing studies on reinforcement learning from human or AI feedback have focused on semantic rewards at the utterance level.
Approach:	They propose a multi-reward RLAIF framework for speech-in/speech-out dialogue systems . they combine semantic, audio-quality, and emotion-consistency rewards .
Outcome:	The proposed framework improves speech-in/speech-out dialogue system quality . it combines semantic, audio-quality, and emotion-consistency rewards . the proposed framework is available to download from the cdc.

Reinforced Efficient Reasoning via Semantically Diverse Exploration (2026.acl-long)

Copied to clipboard

Ziqi Zhao, Zhaochun Ren, Jiahong Zou, Liu Yang, Zhiwei Xu, Xuri Ge, Zhumin Chen, Xinyu Ma, Daiting Shi, Shuaiqiang Wang, Dawei Yin, Xin Xin